Data

Sources

We are collecting our Data from the sustainability development report (SDG), the international labour organization (ILOSTAT), the World Bank, Our world in data, the CATO institute, one from Kaggle (disasters: we couldn’t find relevant accessible information from somewhere else) and GitHub. We found different datasets containing useful information in relation with the SDGs. The details about these data and the links are presented in the next question.

Name of the Table	Source
D1_1_SDG	https://dashboards.sdgindex.org/ downloads
D2_2_Unemployment_rate	https://www.ilo.org/ shinyapps/bulkexplorer3/ ?lang=en&segment=indicator &id=UNE_2EAP_SEX_AGE_RT_A
D3_0_GDP_per_capita	https://data.worldbank.org/ indicator/NY.GDP.PCAP.CD
D3_1_Military_expenditure_percent_GDP	https://data.worldbank.org/ indicator/MS.MIL.XPND.GD.ZS
D3_2_Military_expenditure_percent_gov_exp	https://data.worldbank.org/ indicator/MS.MIL.XPND.ZS
D4_0_Internet_usage	https://ourworldindata.org/ grapher/share-of-individuals- using-the-internet
D5_0_Human_freedom_index	https://www.cato.org/ human-freedom-index/2022
D6_0_Disaters	https://www.kaggle.com/ datasets/brsdincer/all-natural- disasters-19002021-eosdis
D7_0_COVID	https://github.com/ owid/covid-19-data/tree/master/public/data
D8_0_Conflicts	https://datacatalog.worldbank.org/ search/dataset/0041070/ Global-Spread-of-Conflict-by- Country-and-Population

Description

During the wrangling process: we add data to our table (D1_1_SDG) based on different other datasets and match them based on the country, the country code, and the year. The table below shows all our 9 databases that we merge to have our final table for the analysis, as well as each variable of interest that we keep.

Table Name	Variable Name	Explanation	# obs before cleaning	# obs after cleaning
In all database	code	Country code (ISO)
	country	Name of the country
	year	Year of the observation (2000-2022)
D1_1_ SDG	overallscore	Overall score on all 17 SDGs (the score are % of achievement of the goals determined by the UN based on several indicators)	4140 observations of 120 variables	3818 observations of 21 variables
	goal1:goal17	Score on each SDG except SDG 14 (16 variables)
	population	Number of people living in the country
D2_2_ Unemployment_rate	unemployment.rate	Unemployment rate (% of the population 15 years old and older)	82800 observations of 8 variables	571 observations of 5 variables
D3_0_ GDP_per_capita	GDPpercapita	GDP per capita	266 observations of 68 variables	3818 observations of 4 variables
D3_1_ Military_expenditure_ percent_GDP	MilitaryExpenditure PercentGDP	Military expenditures in percentage of GDP	266 observations of 68 variables	3818 observations of 4 variables
D3_2_ Military_expenditure_ percent_gov_exp	MilitaryExpenditure PercentGovExp	Military expenditures in percentage of government expenditures	266 observations of 68 variables	3818 observations of 4 variables
D4_0_ Internet_usage	internet.usage	Internet usage (% of the population)	6570 observations of 4 variables	3433 observations of 4 variables
D5_0_ Human_freedom_index	region	Part of the world, group of countries (e.g. Eastern Europe, Dub-Saharan Africa, South Asia, etc.)	3465 observations of 141 variables	3339 observations of 18 variables
	hf_score	Human Freedom score = mean of personal freedom (PF) and economic freedom (EF).
	pf_law	Rule of law, mean score of: Procedural justice Civil justice Criminal justice Rule of law (V-Dem)
	pf_security	Security and safety, mean score of: Homicide Disappearances conflicts, terrorism
	pf_movement	Freedom of movement (V-Dem) Freedom of movement (CLD)
	pf_religion	Freedom of religion Religious organizatio repression
	pf_assembly	Civil society entry and exit Freedom of assembly Freedom to form/run political parties Civil society repression
	pf_expression	Direct attacks on the press Media and expression (V-Dem) Media and expression (Freedom House) Media and expression (BTI) Media and expression (CLD)
	pf_identity	Same-sex relationships Divorce Inheritance rights Female genital mutilation
	pf_score	Mean of every PF component score
	ef_government	Government consumption Transfers and subsidies Government investment Top marginal tax rate State ownership of assets
	ef_legal	Judicial independence Impartial courts Protection of property rights Military interference Integrity of the legal system Legal enforcementof contracts Regulatory costs Reliability of police
	ef_money	Money growth Standard deviation of inflation Inflation: Most recent year Freedom to own foreign currency
	ef_trade	Tariffs Regulatory trade barriers Black-market exchange rates Movement of capital and people
	ef_regulation	Credit market regulations Labor market regulations Business regulations
	ef_score	Mean of every EF component score
D6_0_ Disaters	continent	Continents touched by the disasters such as floods, ouragan	14644 observations of 47 variables	2435 observations of 10 variables
	total_deaths	Number of total deaths caused by the disasters
	no_injured	Number of injured people
	no_affected	Number of affected people
	no_homeless	Number of people that lost their home and are now homeless
	total_affected	Sum of people affected (sum of the variables: no_injured, no-affected, no_homeless)
	total_damages	Total of infrastructure damages
D7_0_ COVID	deaths_per_million	Number of people dead due to COVID	349966 observations of 67 variables	501 observations (only between 2020-2022, before no COVID) of 6 variables
	cases_per_million	Number of COVID cases
	stringency	Government Response Stringency Index: composite measure based on 9 response indicators including school closures, workplace closures, and trave
D8_0_ Conflicts	ongoing	Variable coded 1 for more than 25 deaths in intrastate conflict and 0 otherwise according to UCDP/PRIO Armed Conflict Dataset 17.1.	5016 observations of 18 variables	2782 observations of 8 variables
	sum_deaths	Best estimate of deaths in all categories of violence (non-state, one-sided and state-based) recorded by the Uppsala Conflict Data Program in the country based on the UCDP GED dataset (unpublished 2016 data). The location of these events is used for estimating the extent of violence.
	pop_affected	Share of population affected by violence in percentage (0 to 100) measured as described above based on population data from CIESIN, the PRIO-GRID structure as well as UCDP GED.
	area_affected	Area affected by conflict
	maxintensity	Two different intensity levels are coded: minor armed conflicts (1) and wars (2), Takes the max intensity of conflict in the country so that it is coded 2 if there is at least one war (>=1000 deaths in intrastate conflict) during the year. Data from UCDP/PRIO Armed Conflict Dataset 17.1.

Wrangling/cleaning

To accommodate the large scale of the datasets we intended to utilize, we decided to pre-clean each of our datasets before merging them. This allowed us to simplify the process of cleaning our final dataset afterwards.

1. Dataset on SDG

This is our main dataset, that we clean in order to keep the columns containing the following inforamtion: country name, country code, year, population, overall score and the 17 SDGs and the score of each SDG exept for SDG 14 (life under water), because there was 24.1% of missing values. We couldn’t replace them, because the information was always missing for a courty across all years. Since the other SDGs such as life on earth and clean water already treat similar subjects, we decided to delete the variable. Two other goals had some missing values: SDG 1 (end poverty) with 9.04% missing values in 15 different countries and SDG 10 (reduced inequalities) with 10.2% missing values.We decide to keep both SDGs and only remove the countries with no information for the analysis, because it isn’t too much countries and that these SDGs are very important. The column population also contained some missing values, but we found out that it was for aggregate groups of countries, that we removed from the database, since we are interested in the different countries only.

# Import data
D1_0_SDG <- read.csv(here("scripts","data","SDG.csv"), sep = ";")

# Transform -> dataframe
D1_0_SDG <- as.data.frame(D1_0_SDG)

# We only want to keep certain columns: country code, country, year, population, overall SDG score and the scores on each SDG
D1_0_SDG <- D1_0_SDG[,1:22]

# Rename the columns to have our variables
colnames(D1_0_SDG) <- c("code", "country", "year", "population", "overallscore", "goal1", "goal2", "goal3", "goal4", "goal5", "goal6", "goal7", "goal8", "goal9", "goal10", "goal11", "goal12", "goal13", "goal14", "goal15", "goal16", "goal17")

# Transform the SDG overall score into a numeric value
D1_0_SDG[["overallscore"]] <- as.double(gsub(",", ".", D1_0_SDG[["overallscore"]]))

# Function to transform the SDG score into numeric values
makenumSDG <- function(D1_0_SDG) {
  for (i in 1:17) {
    varname <- paste("goal", i, sep = "")
    D1_0_SDG[[varname]] <- as.double(gsub(",", ".", D1_0_SDG[[varname]]))
  }
  return(D1_0_SDG)
}

D1_0_SDG <- makenumSDG(D1_0_SDG)

# Make sure the encoding of the country names are UTF-8
D1_0_SDG$country <- stri_encode(D1_0_SDG$country, to = "UTF-8")

# standardize country names
D1_0_SDG <- D1_0_SDG %>%
  mutate(country = countrycode(country, "country.name", "country.name", custom_match = c("T�rkiye"="Turkey")))

# inspection of missing values
propmissing <- numeric(length(D1_0_SDG))

for (i in 1:length(D1_0_SDG)){
  proportion <- mean(is.na(D1_0_SDG[[i]]))
  propmissing[i] <- proportion
}
propmissing
#>  [1] 0.0000 0.0778 0.0000 0.0778 0.0000 0.0833 0.0000 0.0000 0.0000
#> [10] 0.0000 0.0000 0.0000 0.0000 0.0000 0.0944 0.0000 0.0000 0.0000
#> [19] 0.2278 0.0000 0.0000 0.0000

# Population has some missing values, let's investigate
SDG0 <- D1_0_SDG |> 
  group_by(code) |> 
  select(population) |> 
  summarize(NaPop = mean(is.na(population))) |>
  filter(NaPop != 0)
print(SDG0, n = 180)
#> # A tibble: 14 x 2
#>    code         NaPop
#>    <chr>        <dbl>
#>  1 _Africa          1
#>  2 _E_Euro_Asia     1
#>  3 _E_S_Asia        1
#>  4 _HIC             1
#>  5 _LAC             1
#>  6 _LIC             1
#>  7 _LIC_LMIC        1
#>  8 _LMIC            1
#>  9 _MENA            1
#> 10 _OECD            1
#> 11 _Oceania         1
#> 12 _SIDS            1
#> 13 _UMIC            1
#> 14 _World           1

# Normal to have missing values because not countries but regions so we can drop these observations
D1_0_SDG <- D1_0_SDG %>%
  filter(!str_detect(code, "^_"))

# Now there isn't any more missing values in the variable population and we will see that we have information on 166 countries:
(country_number <- length(unique(D1_0_SDG$country)))
#> [1] 166

# Where do we have missing values in the different goal scores? 
SDG1 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na1 = mean(is.na(goal1)),
            Na2 = mean(is.na(goal2)),
            Na3 = mean(is.na(goal3)),
            Na4 = mean(is.na(goal4)),
            Na5 = mean(is.na(goal5)),
            Na6 = mean(is.na(goal6)),
            Na7 = mean(is.na(goal7)),
            Na8 = mean(is.na(goal8)),
            Na9 = mean(is.na(goal9)),
            Na10 = mean(is.na(goal10)),
            Na11 = mean(is.na(goal11)),
            Na12 = mean(is.na(goal12)),
            Na13 = mean(is.na(goal13)),
            Na14 = mean(is.na(goal14)),
            Na15 = mean(is.na(goal15)),
            Na16 = mean(is.na(goal16)),
            Na17 = mean(is.na(goal17))) |>
  filter(Na1 != 0 | Na2 != 0 | Na3 != 0| Na4 != 0| Na5 != 0| Na6 != 0| Na7 != 0| Na8 != 0| Na9 != 0| Na10 != 0| Na11 != 0| Na12 != 0| Na13 != 0| Na14 != 0| Na15 != 0| Na16 != 0| Na17 != 0)

# Print the counts for each variable
kable(for (col in names(SDG1)[-1]) {
  print(paste(col, "count:", sum(SDG1[[col]] != 0)))
})
#> [1] "Na1 count: 15"
#> [1] "Na2 count: 0"
#> [1] "Na3 count: 0"
#> [1] "Na4 count: 0"
#> [1] "Na5 count: 0"
#> [1] "Na6 count: 0"
#> [1] "Na7 count: 0"
#> [1] "Na8 count: 0"
#> [1] "Na9 count: 0"
#> [1] "Na10 count: 17"
#> [1] "Na11 count: 0"
#> [1] "Na12 count: 0"
#> [1] "Na13 count: 0"
#> [1] "Na14 count: 40"
#> [1] "Na15 count: 0"
#> [1] "Na16 count: 0"
#> [1] "Na17 count: 0"

# We see that there are only missings in 3 SDG scores: 1, 10 and 14 and that when there are missings for a country, it is on all years or none. 

# More investigations of those 3 SDG scores. A lot of countries don't have information on those 3 SDG, should we choose to not analyse these SDGs? 
SDG2 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na1 = mean(is.na(goal1))) |>
  filter(Na1 != 0)

print(table(SDG2$Na1))
#> 
#>  1 
#> 15

length(unique(SDG2$code))/country_number
#> [1] 0.0904

# there are only 9.04% missing values in 15 different countries, goal 1 being "end poverty", we decide to keep it and only remove the countries with no information for the analysis
SDG3 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na10 = mean(is.na(goal10))) |>
  filter(Na10 != 0)

print(table(SDG3$Na10))
#> 
#>  1 
#> 17

length(unique(SDG3$code))/country_number
#> [1] 0.102

# there are only 10.2% missing values in 17 different countries, goal 10 being "reduced inequalities", we decide to keep it and only remove the countries with no information for the analysis
SDG4 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na14 = mean(is.na(goal14))) |>
  filter(Na14 != 0)

print(table(SDG4$Na14))
#> 
#>  1 
#> 40

length(unique(SDG4$code))/country_number
#> [1] 0.241

# there are 24.1% missing values in 40 different countries, goal 14 being "life under water", we decide not to keep it, because other SDG such as life on earth and clean water already treat similar subjects

# Delete SDG14
D1_0_SDG <- D1_0_SDG %>% select(-goal14)

We will be working with different datasets and merge them based on the country code and the year. To make sure the match will work well, we standardize the name of the countries and country code using the countrycode library. In addition, we create a liste of all the country codes contained in the main database in order to filter the other databases. Finally, we complete the database to make sure all the combinations of “country, year” are in the database. The number of rows isn’t changed.

# Standardize country code
D1_0_SDG$code <- countrycode(
  sourcevar = D1_0_SDG$code,
  origin = "iso3c",
  destination = "iso3c",
)

# Create a character vector with all the different country codes
list_country <- c(unique(D1_0_SDG$code))

# Create a dataframe with the list of countries and their respective codes
D1_0_SDG_country_list <- D1_0_SDG %>%
  filter(code %in% list_country) %>%
  select(code, country)

# remove duplicated rows
D1_0_SDG_country_list <- D1_0_SDG_country_list %>%
  select(code, country) %>%
  distinct()

# Complete database to make sure there aren't couples of (year, code) missing
D1_0_SDG <- D1_0_SDG |> complete(code, year)

Here are the first few lines of the cleaned dataset on SDG achievement scores:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

As said, this is now our main dataset. All subsequent datasets will be merged with this dataset. Therefore, for all the following datasets, we want to make sure that we only keep data for the same countries and years as in this dataset. We have a total of 166 countries and the years range from 2000 to 2022.

2. Dataset on Unemployment rate

In this dataset, the initial step involves importing the data. Next, we ensure that the names and codes of the countries are formatted in UTF-8, preventing any discrepancies due to mismatches in country names. Following this, we modify the column names and filter the data to include only the relevant countries and years, specifically the years 2000 to 2022, covering 166 countries from our primary dataset.

D2_1_Unemployment_rate <- read.csv(here("scripts","data","UnemploymentRate.csv")) %>%
  as.data.frame() %>%
  mutate(
    country = iconv(ref_area.label, to = "UTF-8", sub = "byte"),
    country = countrycode(country, "country.name", "country.name"),
    year = time,
    `unemployment rate` = obs_value / 100,
    age_category = classif1.label,
    sex = sex.label
  ) %>%
  select(-ref_area.label, -time, -obs_value, -classif1.label, -sex.label, -source.label, -obs_status.label, -indicator.label) %>%
  merge(D1_0_SDG_country_list[, c("country", "code")], by = "country", all.x = TRUE) %>%
  filter(year >= 2000 & year <= 2022,
         !str_detect(sex, fixed("Male")) & !str_detect(sex, fixed("Female")),
         code %in% D1_0_SDG_country_list$code,
         age_category == "Age (Youth, adults): 15+") %>%
  select(code, country, year, `unemployment rate`) %>%
  distinct()

Here are the first few lines of the cleaned dataset on Unemployment rate:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

3. Dataset on GDP military Expenditures

We have three different databases which contain information on each countries over the years. Each year represent one variable. We want to extract three variables for our analysis: GDP per capita, military expenditures in percentage of the GDP and military expenditures in percentage of government expenditures.

GDPpercapita <-
  read.csv(here("scripts","data","GDPpercapita.csv"), sep = ";")
MilitaryExpenditurePercentGDP <-
  read.csv(here("scripts","data","MilitaryExpenditurePercentGDP.csv"), sep = ";")
MiliratyExpenditurePercentGovExp <-
  read.csv(here("scripts","data","MiliratyExpenditurePercentGovExp.csv"), sep = ";")

After importing the data, we fill in the missing country codes using the column Indicator.Name, because we realized after some manipulations, that some of the country codes were false, but the next column contained the right ones.

fill_code <- function(data){
  data <- data %>%
    mutate(Country.Code = ifelse(!grepl("^[A-Z]{3}$", Country.Code), Indicator.Name, Country.Code))
}

We create a set of functions that we will apply to each database. First, remove the variables that we don’t need, which are the years before 2000. Second, make sure that the values are numeric and rename the year variables (because they all had an “X” before year number). Third, transform the database from wide to long, in order to match the main database. Fourth, transform the year variable into an integer variable and rearrange and rename the columns to match the ones of the other databases. Then, we apply these transformations to the three databases.

# remove the variables that we don't need
remove <- function(data){
  years <- seq(1960, 1999)
  removeyears <- paste("X", years, sep = "")
  data <- data[, !(names(data) %in% c("Indicator.Name", "Indicator.Code", "X", removeyears))]
}

# Make sure that the values are numeric
makenum <- function(data) {
  for (i in 2000:2022) {
    year <- paste("X", i, sep = "")
    data[[year]] <- as.numeric(data[[year]])
  }
  return(data)
}

# Rename years from Xyear 
renameyear <- function(data) {
  for (i in 2000:2022) {
    varname <- paste("X", i, sep = "")
    names(data)[names(data) == varname] <- gsub("X", "", varname)
  }
  return(data)
}

# Transform the database from wide to long
wide2long <- function(data) {
  data <- pivot_longer(data, 
                       cols = -c("Country.Name", "Country.Code"), 
                       names_to = "year", 
                       values_to = "data")
  return(data)
}

# Transform the year variable into an integer variable
yearint <- function(data) {
  data$year <- as.integer(data$year)
  return(data)
}

# Rearrange and rename the columns to match the ones of the other datasets
nameorder <- function(data) {
  colnames(data) <- c("country", "code", "year", "data")
  data <- data %>% select(c("code", "country", "year", "data"))
}

# One function that contains all the others
cleanwide2long <- function(data){
  data <- fill_code(data)
  data <- remove(data)
  data <- makenum(data)
  data <- renameyear(data)
  data <- wide2long(data)
  data <- yearint(data)
  data <- nameorder(data)
}

# Apply function to three database
GDPpercapita <- cleanwide2long(GDPpercapita)
MilitaryExpenditurePercentGDP <- cleanwide2long(MilitaryExpenditurePercentGDP)
MiliratyExpenditurePercentGovExp <- cleanwide2long(MiliratyExpenditurePercentGovExp)

We rename the colums with the main information, standardize the country code and remove the countries that are not in our main database. We see that all the 166 countries are there.

# Rename the data columns to have the right name
GDPpercapita <- GDPpercapita %>%
  rename(GDPpercapita = data)

MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>%
  rename(MilitaryExpenditurePercentGDP = data)

MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>%
  rename(MiliratyExpenditurePercentGovExp = data)

# Standardize the country code
GDPpercapita$code <- countrycode(
  sourcevar = GDPpercapita$code,
  origin = "iso3c",
  destination = "iso3c",
)

MilitaryExpenditurePercentGDP$code <- countrycode(
  sourcevar = MilitaryExpenditurePercentGDP$code,
  origin = "iso3c",
  destination = "iso3c",
)

MiliratyExpenditurePercentGovExp$code <- countrycode(
  sourcevar = MiliratyExpenditurePercentGovExp$code,
  origin = "iso3c",
  destination = "iso3c",
)

# Remove the obervations of countries that aren't in our main dataset on SDGs: 
GDPpercapita <- GDPpercapita %>% filter(code %in% list_country)
length(unique(GDPpercapita$code))
#> [1] 166

MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>% filter(code %in% list_country)
length(unique(MilitaryExpenditurePercentGDP$code))
#> [1] 166

MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>% filter(code %in% list_country)
length(unique(MiliratyExpenditurePercentGovExp$code))
#> [1] 166

# There are only 157 countries that are both in the main SDG dataset and in these 3 datasets. But we suspect that some of the missing countries were in the database but not rightly matched
list_country_GDP <- c(unique(GDPpercapita$code))
(missing <- setdiff(list_country, list_country_GDP))
#> character(0)

# 1. Bahamas was in the database but instead of the code "BHS" there is "The"
# 2. "COD" "Dem. Rep."
# 3. "COG" "Rep"
# 4. "EGY" "Arab Rep."
# 5. "GMB" "The"
# 6. "IRN" "Islamic Rep."
# 7. "KOR" "Rep."
# 8. "VEN" "RB"
# 9. "YEM" "Rep."
# We remark that the code is in another column of the initial database: "Indicator.Name"
# We go back to the initial database and before cleaning it we put the right codes

# After rerunning the code we see that we have all our 166 countries from the initial dataset 

We run a first round of investigation of the missing values and find that we have 16.4% for MiliratyExpenditurePercentGovExp, 12.9% for MilitaryExpenditurePercentGDP and 1.31% for GDPpercapita.

# What is the percentage of missing values in these 3 datasets?
mean(is.na(MiliratyExpenditurePercentGovExp$MiliratyExpenditurePercentGovExp))
#> [1] 0.164
mean(is.na(MilitaryExpenditurePercentGDP$MilitaryExpenditurePercentGDP))
#> [1] 0.129
mean(is.na(GDPpercapita$GDPpercapita))
#> [1] 0.0131

# 16.4% for MiliratyExpenditurePercentGovExp, 12.9% for MilitaryExpenditurePercentGDP and 1.31% for GDPpercapita

GDP per capita

For GDPpercapita, only two countries (SOM and SSD) have a lot of missing values and in total 11 countries countries have missing values.

####### Investigate missing values in GDPpercapita ######
GDPpercapita1 <- GDPpercapita %>%
  group_by(code) %>%
  summarize(NaGDP = mean(is.na(GDPpercapita))) %>%
  filter(NaGDP != 0)
print(GDPpercapita1, n = 180)
#> # A tibble: 11 x 2
#>    code   NaGDP
#>    <chr>  <dbl>
#>  1 AFG   0.130 
#>  2 BTN   0.0435
#>  3 CUB   0.0870
#>  4 LBN   0.0435
#>  5 SOM   0.565 
#>  6 SSD   0.652 
#>  7 STP   0.0435
#>  8 SYR   0.0870
#>  9 TKM   0.0870
#> 10 VEN   0.304 
#> 11 YEM   0.130

# Only SOM and SSD have a lot of missings and in total 11 countries with missings

We plot the evolution of GDPpercapita avec the years for each country containing missing values and distinguish the percentage of missing values with colors.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
filtered_data_GDP <- GDPpercapita %>%
  filter(code %in% GDPpercapita1$code)

filtered_data_GDP <- filtered_data_GDP %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(GDPpercapita))) %>%
  ungroup()

# Look at the evolution over the years for the countries that have missing values
Evol_Missing_GDP <- ggplot(data = filtered_data_GDP) +
  geom_point(aes(x = year, y = GDPpercapita, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of GDP per capita over time", x = "Year", y = "GDP per capita") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "30-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 4)

print(Evol_Missing_GDP)

# We decide not to use SSD SOM and VEN since there are more than 30% missing

For the countries with less than 30% of missing values and a linear evolution in time, we fill the missing values using linear interpolation.

####### Fill in missing values in GDPpercapita ######

# Almost all have a linear evolution over time, we fill in the missing values based on the lines

# AFG, BTN, CUB, STP and TKM are easy with only one line
list_code <- c("AFG", "BTN", "CUB", "STP", "TKM")

for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- GDPpercapita %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$GDPpercapita)
  
  # Update the original dataset with the interpolated values
  GDPpercapita[GDPpercapita$code == i, "GDPpercapita"] <- interpolated_data
}

# SYR and YEM : we fit 2 lines to fill the values

# LBN: weird at the end, we don't fill the missing value for now

Military expenditures in percentage of GDP

For MilitaryExpenditurePercentGDP, 12 countries have 100% of missing values. We further investigate and keep them for now, knowing that some of these coutries may also have many missing values in the other databases when wee merge everything and will be dropped later.

##### Investigate missing values in MilitaryExpenditurePercentGDP #####
MilitaryExpenditurePercentGDP1 <- MilitaryExpenditurePercentGDP %>%
  group_by(code) %>%
  summarize(NaMil1 = round(mean(is.na(MilitaryExpenditurePercentGDP)),3)) %>%
  filter(NaMil1 != 0)

print(table(MilitaryExpenditurePercentGDP1$NaMil1))
#> 
#> 0.043 0.087  0.13 0.174 0.217 0.261 0.304 0.348 0.391 0.522 0.565 
#>     4     2     7     6     3     3     3     2     1     2     2 
#> 0.739 0.783     1 
#>     1     1    12

# 100% missing: a lot! 12 countries

We plot the evolution of MilitaryExpenditurePercentGDP along the years for each country containing missing values and distinguish the percentage of missing values with colors.

# Create a dataframe that only have the coutnries with missing values and 
# add a column which contains the % of missings for each country
filtered_data_Mil1 <- MilitaryExpenditurePercentGDP %>%
  filter(code %in% MilitaryExpenditurePercentGDP1$code)

filtered_data_Mil1 <- filtered_data_Mil1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>%
  ungroup()

# Look at evolution over the years
Evol_Missing_Mil1 <- ggplot(data = filtered_data_Mil1) +
  geom_line(aes(x = year, y = MilitaryExpenditurePercentGDP, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Military expenditure in % of GDP over time", x = "Years from 2000 to 2022", y = "GDP per capita") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 5) +
  theme(strip.text = element_text(size = 6)) +
  scale_x_continuous(breaks = NULL) +
  scale_y_continuous(breaks = NULL)

print(Evol_Missing_Mil1)

For the countries with less than 30% of missing values and a linear evolution in time, we fill the missing values using linear interpolation.

# Try to fill the missings if %missings < 30%

##### Fill in missing values in MilitaryExpenditurePercentGDP #####

# "AFG", "BDI", "BEN", "CAF", "CIV", "COD", "GAB", "GMB", "KAZ", "LBN", "LBR", "MNE", "MRT", "NER", "TKJ", "TTO", "ZMB"
# <30% missing and linear (17)
list_code <- c("AFG", "BDI", "BEN", "CAF", "CIV", "COD", "GAB", "GMB", "KAZ", "LBN", "LBR", "MNE", "MRT", "NER", "TKJ", "TTO", "ZMB")

for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- MilitaryExpenditurePercentGDP %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$MilitaryExpenditurePercentGDP)
  
  # Update the original dataset with the interpolated values
  MilitaryExpenditurePercentGDP[MilitaryExpenditurePercentGDP$code == i, "MilitaryExpenditurePercentGDP"] <- interpolated_data
}

# "BIH", "COG", "IRQ", "MMR", "SDN", "TCD", "TGO", "ZWE"
# <30% missing but not linear (keep but we will see later) (8)

# Others have too much missing (24)

Military expenditures in percentage of governement expenditures

For MilitaryExpenditurePercentGovExp, 17 countries have 100% of missing values. We further investigate and keep them for now, knowing that some of these coutries may also have many missing values in the other databases when wee merge everything and will be dropped later.

##### Investigate missing values in MilitaryExpenditurePercentGovExp #####

MiliratyExpenditurePercentGovExp1 <- MiliratyExpenditurePercentGovExp %>%
  group_by(code) %>%
  summarize(NaMil2 = round(mean(is.na(MiliratyExpenditurePercentGovExp)),3)) %>%
  filter(NaMil2 != 0)

print(table(MiliratyExpenditurePercentGovExp1$NaMil2))
#> 
#> 0.043 0.087  0.13 0.174 0.217 0.261 0.304 0.348 0.391 0.478 0.522 
#>     5     3     5     4     5     4     4     2     1     1     2 
#> 0.565 0.609 0.783     1 
#>     2     1     1    17

# 100% missing: a lot ! 17 countries

We plot the evolution of MilitaryExpenditurePercentGovExp along the years for each country containing missing values and distinguish the percentage of missing values with colors.

# Create a dataframe that only have the coutnries with missing values and add a column which contains the % of missings for each country
filtered_data_Mil2 <- MiliratyExpenditurePercentGovExp %>%
  filter(code %in% MiliratyExpenditurePercentGovExp1$code)

filtered_data_Mil2 <- filtered_data_Mil2 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(MiliratyExpenditurePercentGovExp))) %>%
  ungroup()

# Look at evolution over the years
Evol_Missing_Mil2 <- ggplot(data = filtered_data_Mil2) +
  geom_line(aes(x = year, y = MiliratyExpenditurePercentGovExp, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Military expenditure in % of government expenditures over time", x = "Year from 2000 to 2022", y = "GDP per capita") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 5) +
  theme(strip.text = element_text(size = 6)) +
  scale_x_continuous(breaks = NULL) +
  scale_y_continuous(breaks = NULL)

print(Evol_Missing_Mil2)

For the countries with less than 30% of missing values and a linear evolution in time, we fill the missing values using linear interpolation.

# Try to fill the missings if %missings < 30%

##### Fill in missing values in MilitaryExpenditurePercentGovExp #####

# "AFG", "ARM", BEN", "BIH", "BLR", COG", "ECU", "GAB", "GMB", "KAZ", "LBN", "LBR", "MNE", "MWI", "NER", "TTO", "UKR", "ZMB" <30% missing and linear (18)
list_code <- c("AFG", "ARM", "BEN", "BIH", "BLR", "COG", "ECU", "GAB", "GMB", "KAZ", "LBN", "LBR", "MNE", "MWI", "NER", "TTO", "UKR", "ZMB")

for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- MiliratyExpenditurePercentGovExp %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$MiliratyExpenditurePercentGovExp)
  
  # Update the original dataset with the interpolated values
  MiliratyExpenditurePercentGovExp[MiliratyExpenditurePercentGovExp$code == i, "MiliratyExpenditurePercentGovExp"] <- interpolated_data
}

# "BDI", "IRQ"
# 2 lines (2)

# "CAF", "MMR", "SDN", "TCD", "TGO", "TJK" 
# <30% missing but not linear (keep but we will see later) (6)

# Others have too much missing (31) -> very much maybe we will have to drop this variable for our analysis

We now look again at the percentage of missing values for the trhee databases: 14.49% for MiliratyExpenditurePercentGovExp, 11.6% for MilitaryExpenditurePercentGDP and 1.07% for GDPpercapita

# And now, What is the percentage of missing values in these 3 datasets?
mean(is.na(MiliratyExpenditurePercentGovExp$MiliratyExpenditurePercentGovExp))
#> [1] 0.149
mean(is.na(MilitaryExpenditurePercentGDP$MilitaryExpenditurePercentGDP))
#> [1] 0.116
mean(is.na(GDPpercapita$GDPpercapita))
#> [1] 0.0107

# Standardize names for merge
D3_1_GDP_per_capita <- GDPpercapita
D3_2_Military_Expenditure_Percent_GDP <- MilitaryExpenditurePercentGDP
D3_3_Miliraty_Expenditure_Percent_Gov_Exp <- MiliratyExpenditurePercentGovExp

Here are the first few lines of the cleaned dataset of GDP per capita:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

Here are the first few lines of the cleaned dataset of military expenditures in percentage of GDP:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

Here are the first few lines of the cleaned dataset of military expenditures in percentage of government expenditures:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

4. Dataset on internet usage

To prepare the dataset on internet usage in the world to be merge with the other data, we first, import the data. Then, we keep only the year that we are interested in (2000 to 2022). We also rename the column and keep only the country that match the list of the countries in the main dataset on the SDG.

D4_0_Internet_usage <- read.csv(here("scripts", "data", "InternetUsage.csv")) %>%
  filter(Year >= 2000, Year <= 2022) %>%
  rename(
    code = Code,
    country = Entity,
    year = Year,
    internet_usage = Individuals.using.the.Internet....of.population.
  ) %>%
  mutate(internet_usage = internet_usage / 100) %>%
  filter(code %in% list_country)

Here are the first few lines of the cleaned dataset of internet usage:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

5. Dataset on human freedom index

After importing the data from the CATO Institute website, we noticed that even if the file was called “Human Freedom Index 2022”, the available observations were only going from 2000 up to 2020. We have decided first to modify it in order to match our other datasets, by renaming/encoding/standardizing the columns containing the country names.

data <- read.csv(here("scripts", "data", "human-freedom-index-2022.csv"))

#data in tibble 
datatibble <- tibble(data)

# Rename the column countries into country to match the other datbases
names(datatibble)[names(datatibble) == "countries"] <- "country"

# Make sure the encoding of the country names are UTF-8
datatibble$country <- iconv(datatibble$country, to = "UTF-8", sub = "byte")

# standardize country names
datatibble <- datatibble %>%
  mutate(country = countrycode(country, "country.name", "country.name"))

Once done, we could verify which countries were or were not present between these observations and our main SDG dataset. We have decided to keep the ones that were matching between the two datasets.

# Merge by country name
datatibble <- datatibble %>%
  left_join(D1_0_SDG_country_list, by = "country")

# Keep only the countries that are in our main dataset

datatibble <- datatibble %>% filter(code %in% list_country)
(length(unique(datatibble$code)))
#> [1] 159

# See which ones are missing

list_country_free <- c(unique(datatibble$code))
(missing <- setdiff(list_country, list_country_free))
#> [1] "AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB"

# Turkey was missing but present in the initial database (it was a problem when stadardizing the country names of D1_0SDG_country_list that we corrected) and the other missing countries are:"AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB" 

D5_0_Human_freedom_index <- datatibble

Then, we noticed that there were a lot of columns that were not important for us, as we had 141 variables taken into account. So we have decided to keep the ones that refers to the countries informations (such as code, year, ..) and their human freedom scores per category (pf for personnal freedom, ef for economical freedom).

# erasing useless columns to keep only the general ones. 

D5_0_Human_freedom_index <- select(D5_0_Human_freedom_index, year, country, region, hf_score, pf_rol, pf_ss, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, pf_score, ef_government, ef_legal, ef_money, ef_trade, ef_regulation, ef_score, code)

D5_0_Human_freedom_index <- D5_0_Human_freedom_index %>%
  rename(
    pf_law = names(D5_0_Human_freedom_index)[5],      # Renames the 5th column to "pf_law"
    pf_security = names(D5_0_Human_freedom_index)[6]  # Renames the 6th column to "pf_security"
  )

After renaming the columns pf_law/security for comprehension purpose, we have made a heatmap that helped us to visualize in our data where were the NA values per country and per variable in percentage, ordered by countries having at least one variable with more than 50% of missing. This gave us a better comprehension of the distribution of the NA values. Finally, we noticed that only 13 countries of our dataset had more than 50% of missing values for one or more variables.

##### VISUALIZATION ##### 

#Find NA percentage per country per variable 

na_percentage_by_country <- D5_0_Human_freedom_index %>%
  group_by(country) %>%
  summarise(across(everything(), ~mean(is.na(.))*100))

na_long <- na_percentage_by_country %>%
  pivot_longer(
    cols = -country,
    names_to = "Variable",
    values_to = "NA_Percentage"
  )

overall_na_percentage <- na_long %>%
  group_by(Variable) %>%
  summarize(Avg_NA_Percentage = mean(NA_Percentage, na.rm = TRUE)) %>%
  arrange(desc(Avg_NA_Percentage))

print(overall_na_percentage)
#> # A tibble: 18 x 2
#>    Variable      Avg_NA_Percentage
#>    <chr>                     <dbl>
#>  1 ef_money                 10.4  
#>  2 ef_trade                 10.4  
#>  3 ef_score                 10.4  
#>  4 hf_score                 10.4  
#>  5 pf_score                 10.4  
#>  6 ef_regulation             9.49 
#>  7 ef_government             2.91 
#>  8 ef_legal                  1.71 
#>  9 pf_law                    1.44 
#> 10 pf_identity               0.299
#> 11 code                      0    
#> 12 pf_assembly               0    
#> 13 pf_expression             0    
#> 14 pf_movement               0    
#> 15 pf_religion               0    
#> 16 pf_security               0    
#> 17 region                    0    
#> 18 year                      0

# Order the countries with between 50 and 100 of NA values 

na_long <- na_long %>%
  group_by(country) %>%
  mutate(Count_NA_50_100 = sum(NA_Percentage >= 50 & NA_Percentage <= 100, na.rm = TRUE)) %>%
  ungroup() %>%
  arrange(desc(Count_NA_50_100))

# Now, visualize

heatmap_ordered <- ggplot(na_long, aes(x = reorder(country, -Count_NA_50_100), y = Variable)) +
  geom_tile(aes(fill = NA_Percentage), colour = "white") +
  scale_fill_gradient(low = "white", high = "red") +
  theme_minimal() +
  labs(
    title = "Heatmap of NA Percentages per Country and Variable",
    x = "Country",
    y = "Variable",
    fill = "NA Percentage"
  ) +
  theme(
    axis.text.x = element_text(angle = 90, hjust = 1),
    axis.text.y = element_text(size = 7)
  )
print(heatmap_ordered)

###### END VISUALIZATION ######

#Checking the number of variables per countries where NA values percentage >=50%

country_na_count <- na_long %>%
  filter(NA_Percentage >= 50) %>%
  group_by(country) %>%
  summarise(Count_NA_50_100 = n()) %>%
  arrange(desc(Count_NA_50_100))

print(country_na_count)
#> # A tibble: 13 x 2
#>    country  Count_NA_50_100
#>    <chr>              <int>
#>  1 Comoros                8
#>  2 Djibouti               8
#>  3 Somalia                8
#>  4 Belarus                6
#>  5 Guinea                 6
#>  6 Iraq                   6
#>  7 Laos                   6
#>  8 Sudan                  6
#>  9 Bhutan                 5
#> 10 Liberia                5
#> 11 Bahamas                1
#> 12 Belize                 1
#> 13 Brunei                 1

When discussing between us, we came to the conclusion that among these 13 countries, a great part of them were not going to be selected because they had a lot of missing values in our main dataset too. Therefore, we have decided to merge this data with the other datasets and finish the cleaning after.

6. Dataset on Disasters

# Natural Disasters

# Read the CSV data file into a data frame named "Disasters"
Disasters <- read.csv(here("scripts","data","Disasters.csv"))

# Convert "Disasters" into a data frame (if it's not already)
Disasters <- as.data.frame(Disasters)

# Select specific columns of interest from the "Disasters" data frame
Disasters <- Disasters %>%
  select(Year, Country, ISO, Location, Continent, Disaster.Subgroup, Disaster.Type, Total.Deaths, No.Injured, No.Affected, No.Homeless, Total.Affected, Total.Damages...000.US..)

# Rearrange the columns, changed the type of data, renamed the columns
Rearanged_Disasters <- Disasters %>%
  filter(Year >= 2000 & Year <= 2022) %>%
  mutate(
    code = as.character(ISO),
    country = as.character(Country),
    year = as.integer(Year),
    continent = as.character(Continent),
    disaster.subgroup = as.character(Disaster.Subgroup),
    disaster.type = as.character(Disaster.Type),
    location = as.character(Location),
    total.deaths = as.numeric(Total.Deaths),
    no.injured = as.numeric(No.Injured),
    no.affected = as.numeric(No.Affected),
    no.homeless = as.numeric(No.Homeless),
    total.affected = as.numeric(Total.Affected),
    total.damages = as.numeric(Total.Damages...000.US..)
  )


# Group the data by "year", "code", "country" and "continent" and summarize the data
Disasters <- Rearanged_Disasters %>%
  group_by(year,code, country, continent) %>%
  summarize(
    total_deaths = sum(total.deaths, na.rm = TRUE),
    no_injured = sum(no.injured, na.rm = TRUE),
    no_affected = sum(no.affected, na.rm = TRUE),
    no_homeless = sum(no.homeless, na.rm = TRUE),
    total_affected = sum(total.affected, na.rm = TRUE),
    total_damages = sum(total.damages, na.rm = TRUE)
  ) 

# Select specific columns from the summarized data and arrange the data by specified columns
D6_0_Disasters <- Disasters %>%
  select(code, country, year, continent, total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages) %>%
  arrange(code, country, year, continent)

# Keep only the countries that are in our main dataset

D6_0_Disasters <- D6_0_Disasters %>% filter(code %in% list_country)
length(unique(D6_0_Disasters$code))
#> [1] 163

# See which countries are missing
list_country_disasters <- c(unique(D6_0_Disasters$code))
(missing <- c(missing,setdiff(list_country, list_country_disasters)))
#>  [1] "AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB" "BHR" "BRN" "MLT"

# Weird 3 missing: BHR, BRN and MLT

7. Dataset on COVID

This dataset contains information on the COVID19 pandemic between 2020 and 2022. The observation are by year, month, day. After importing the database, we transform the date in format YYYY-MM-DD in order to only keep the year.

#Import data
COVID <- read.csv(here("scripts","data","COVID.csv"))

# Only keep the variables that we are interested in
COVID <- COVID[,c("iso_code", "location", "date", "new_cases_per_million", "new_deaths_per_million", "stringency_index")]

# Transform the dates ("YYYY-MM-DD") into years ("YYYY") and integers
COVID$date <- as.integer(year(COVID$date))

We perform a first round of investigation of the missing values before aggregating the values by year. We begin with the variables “cases per million” and “deaths per million”: seeing that for each country, we have either only missing values, either a very low percentage of missing values (~1%), we can compute the sum over each year and ignore the missing values without altering the data. Indeed, where al the values are missing, the computation will return a NA. We then look at the “stringency” variable and we have 3 scenarios:

1. ~20% missings: we ignore missing values when computing the mean to have an idea of stringency each year (because we compute the mean stringency over the year, if some days are missing, it is not a problem, it can not evoluate that fast).

2. all are missing : we can ignore the missing values when computing the mean, because it will still return a missing value

3. almost all are missing: here the mean doesn’t make sense -> we will replace the values by NAs to be coherent. The countries with this issues are: ERI, GUM, PRI and VIR. We verify if they are in our main dataset and since none of these countries are, we can ignore the issue, the lines will be remove later anyway.

We aggregate the observations of all days of a year in one observation per country using the mean.

# Investigate missing values before aggregating
COVID1 <- COVID %>%
  group_by(iso_code) %>%
  summarize(NaCOVID = round(mean(is.na(new_cases_per_million)),3)) %>%
  filter(NaCOVID != 0)

print(table(COVID1$NaCOVID))
#> 
#> 0.001 0.002 0.003 0.004 0.012 0.109     1 
#>    33     6     2     5     1     1     9

COVID2 <- COVID %>%
  group_by(iso_code) %>%
  summarize(NaCOVID = round(mean(is.na(new_deaths_per_million)),3)) %>%
  filter(NaCOVID != 0)

print(table(COVID2$NaCOVID))
#> 
#> 0.001 0.002 0.004  0.11     1 
#>    32     1     2     1     9

# We see that for each country, we have either only missing values, either a very low % of missing ~1% -> we can compute the sum over each year and ignore the missing values withoutaltering the data
COVID3 <- COVID %>%
  group_by(iso_code) %>%
  summarize(NaCOVID = round(mean(is.na(stringency_index)), 3)) %>%
  filter(NaCOVID != 0)

print(table(COVID3$NaCOVID))
#> 
#>  0.13 0.186 0.198  0.21 0.986     1 
#>     1     1     1   178     4    70

# Here we have 3 scenarios: 
# 1) ~20% missings -> ok to ignore missings when computing the mean to have an idea of stringency each year
# 2) all missings -> ok to ignore missings when computing the mean because it will still return a missing value
# 3) almost all are missing: here the mean doesn't make sense -> we will replace the values by NAs to be coherent. The countries with this issues are: ERI, GUM, PRI and VIR, we want to see if these countries are in our main dataset
issue_list <- c("ERI", "GUM", "PRI", "VIR")
is.element(issue_list, list_country)
#> [1] FALSE FALSE FALSE FALSE

# Since non of these countries are in the main SDG database, we can ignore the issue, the lines will be remove later anyway

# Aggregate the observation of all days of a year in one observation per country
COVID <- COVID %>%
  group_by(location, date) %>%
  mutate(
    cases_per_million = sum(new_cases_per_million, na.rm = TRUE),
    deaths_per_million = sum(new_deaths_per_million, na.rm = TRUE),
    stringency = mean(stringency_index, na.rm = TRUE)
  )%>%
  ungroup()

Now that all the variables of interest are aggregated by year, we remove all the variables that we don’t need and rename all the remaining variables to match the main dataset.

# Only have 1 obs per country per year
COVID <- COVID %>%
  group_by(location, date) %>%
  distinct(date, .keep_all = TRUE) %>%
  ungroup()

# Remove the variable that have the information for every day and only keep those by year
COVID <- COVID %>% select(-c(new_cases_per_million, new_deaths_per_million, stringency_index))

# Rename the variables
colnames(COVID) <- c("code", "country", "year", "cases_per_million", "deaths_per_million", "stringency")

We remove the years that exceed 2022, we make sure that the country codes are all iso codes with 3 letters (we observe that sometimes they are preceded by “OWID_”) and we standardize the country codes.

# Remove the years after 2022 to match our main database 
COVID <- COVID[COVID$year <= 2022, ]

# Make sure the country codes are all iso codes with 3 letters (we observe that sometimes they are preceded by "OWID_")
COVID$code <- gsub("OWID_", "", COVID$code)

# Standardize the country code
COVID$code <- countrycode(
  sourcevar = COVID$code,
  origin = "iso3c",
  destination = "iso3c",
)

We remove the observations of countries that aren’t in our main dataset on SDGs and find that all the 166 countries that we have in the main SDG dataset are also in this one.

# Remove the observations of countries that aren't in our main dataset on SDGs: 
COVID <- COVID %>% filter(code %in% list_country)
length(unique(COVID$code))
#> [1] 166

# All the 166 countries that we have in the main SDG dataset are also in this one.

We perform a second round of missing values investigation and find out that there are no missing values except for the stringency, where there are 4.19%. Either all values are missing for one country, or 50% are missing, so these 7 countries won’t be included when analyzing the effect of stringency on the SDG scores.

##### Investigation of the missing values #####
mean(is.na(COVID$cases_per_million))
#> [1] 0
mean(is.na(COVID$deaths_per_million))
#> [1] 0
mean(is.na(COVID$stringency))
#> [1] 0.0419

# No missing values except in for the stringency, where there are 4.19% 
COVID4 <- COVID %>%
  group_by(code) %>%
  summarize(NaCOVID = mean(is.na(stringency))) %>%
  filter(NaCOVID != 0)
print(COVID4, n = 300)
#> # A tibble: 7 x 2
#>   code  NaCOVID
#>   <chr>   <dbl>
#> 1 ARM       1  
#> 2 COM       1  
#> 3 MDV       1  
#> 4 MKD       1  
#> 5 MNE       1  
#> 6 NAM       0.5
#> 7 STP       1

# Either all values are missing for one country, or 50% are missing, so these 7 countries 
# won't be included when analysing the effect of stringency
D7_0_COVID <- COVID

Here are the first few lines of the cleaned dataset on COVID19:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

8. Dataset on Conflicts

#Conflicts

# Read the Excel data file into a data frame named "Conflicts"
Conflicts <- read.csv(here("scripts","data","Conflicts.csv"))

# Convert "Conflicts" into a data frame (if it's not already)
Conflicts <- as.data.frame(Conflicts)

# Select specific columns of interest from the "Conflicts" data frame
Conflicts <- Conflicts %>%
  select(year, country, ongoing, gwsum_bestdeaths, pop_affected, peaceyearshigh, area_affected, maxintensity, maxcumulativeintensity)

# Filter rows based on the "year" column
#Note: our data set has no more information about conflicts per country from 2016. As we consider conflicts as events, we will only take into account results between 2000 and 2016.
Rearanged_Conflicts <- Conflicts %>%
  filter(year >= 2000 & year <= 2022)%>%
  mutate(
    ongoing = as.integer(ongoing),
    country = as.character(country),
    year = as.integer(year),
    gwsum_bestdeaths = as.numeric(gwsum_bestdeaths),
    pop_affected = as.numeric(pop_affected),
    area_affected = as.numeric(area_affected),
    maxintensity = as.numeric(maxintensity),
    )

# Group the data by "year", "country" and summarize the data
Conflicts <- Rearanged_Conflicts %>%
  group_by(year, country) %>%
  summarize(
    ongoing = sum (ongoing, na.rm = TRUE),
    sum_deaths = sum(gwsum_bestdeaths, na.rm = TRUE),
    pop_affected = sum(pop_affected, na.rm = TRUE),
    area_affected = sum(area_affected, na.rm = TRUE),
    maxintensity = sum(maxintensity, na.rm = TRUE),
  )
    
# Select specific columns from the summarized data and arrange the data by specified columns
conflicts <- Conflicts %>%
  select(country, year, ongoing, sum_deaths, pop_affected, area_affected, maxintensity) %>%
  arrange(country, year)


# Print the summary of the "Rearanged_Conflicts" data frame
#summary(conflicts)

# Make sure the encoding of the country names are UTF-8
conflicts$country <- iconv(conflicts$country, to = "UTF-8", sub = "byte")

# standardize country names
conflicts <- conflicts %>%
  mutate(country = countrycode(country, "country.name", "country.name"))

# Merge by country name
conflicts <- conflicts %>%
  left_join(D1_0_SDG_country_list, by = "country")

#Rearrange the data
conflicts <- conflicts %>%
  select(code, country, year, ongoing, sum_deaths, pop_affected, area_affected, maxintensity) %>%
  arrange(code, country, year)

# Keep only the countries that are in our main dataset

D8_0_Conflicts <- conflicts %>% filter(code %in% list_country)
(length(unique(conflicts$code)))
#> [1] 166

# See which ones are missing

list_country_conflicts <- c(unique(conflicts$code))
(missing <- c(missing, setdiff(list_country, list_country_conflicts)))
#>  [1] "AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB" "BHR" "BRN" "MLT"
#> [11] "BLR"

# Only one country missing that wasn't in the inital conflicts database: BLR

Merge data

By merging our eight pre-cleaned datasets, we create a final database.

# merge D1_0_SDG with D2_1_Unemployment_rate 
D2_1_Unemployment_rate$country <- NULL
merge_1_2 <- D1_0_SDG |> left_join(D2_1_Unemployment_rate, join_by(code, year))

# merge merge_1_2 with D3_1_GDP_per_capita, D3_2_Military_Expenditure_Percent_GDP and D3_3_Miliraty_Expenditure_Percent_Gov_Exp
D3_1_GDP_per_capita$country <- NULL
merge_12_3 <- merge_1_2 |> left_join(D3_1_GDP_per_capita, join_by(code, year))

D3_2_Military_Expenditure_Percent_GDP$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_2_Military_Expenditure_Percent_GDP, join_by(code, year)) 

D3_3_Miliraty_Expenditure_Percent_Gov_Exp$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_3_Miliraty_Expenditure_Percent_Gov_Exp, join_by(code, year)) 

# merge merge_12_3 with D4_0_Internet_usage 
D4_0_Internet_usage$country <- NULL
merge_123_4 <- merge_12_3 |> left_join(D4_0_Internet_usage, join_by(code, year)) 

# merge merge_123_4 with D5_0_Human_freedom_index
D5_0_Human_freedom_index$country <- NULL
merge_1234_5 <- merge_123_4 |> left_join(D5_0_Human_freedom_index, join_by(code, year)) 

# merge merge_1234_5 with D_6_0_Disasters
D6_0_Disasters$country <- NULL
merge_12345_6 <- merge_1234_5 |> left_join(D6_0_Disasters, join_by(code, year)) 

# merge merge_12345_6 with D7_0_COVID
D7_0_COVID$country <- NULL
D7_0_COVID <- D7_0_COVID |> distinct(code, year, .keep_all = TRUE)
merge_123456_7 <- merge_12345_6 |> left_join(D7_0_COVID, join_by(code, year)) 

# merge merge_123456_7 with D8_0_Conflicts
D8_0_Conflicts$country <- NULL
all_Merge <- merge_123456_7 |> left_join(D8_0_Conflicts, join_by(code, year)) 

# Filter to delete the countries that were missing from some of our databases
all_Merge <- all_Merge %>% filter(!code %in% missing)

Cleaning of the final database

We replace the NAs of the COVID columns by 0 (because we don’t have real missing, only introduced by merging for the years before COVID).

# Replace the NAs of the COVID columns by 0 (because we don't have real missing, only introduced by merging for the years before COVID)
all_Merge <- all_Merge %>%
  mutate(
    cases_per_million = ifelse(is.na(cases_per_million), 0, cases_per_million),
    deaths_per_million = ifelse(is.na(deaths_per_million), 0, deaths_per_million),
    stringency = ifelse(is.na(stringency), 0, stringency)
  )

Since we took the information on the continent and region from databases that are not the main one, we complete these inforamtion for the whole final dataset.

# Complete the values of continent and region
all_Merge <- all_Merge %>%
  group_by(country) %>%
  mutate(continent = ifelse(is.na(continent), first(na.omit(continent)), continent)) %>%
  ungroup()

all_Merge <- all_Merge %>%
  group_by(country) %>%
  mutate(region = ifelse(is.na(region), first(na.omit(region)), region)) %>%
  ungroup()

We order the database, beginning by the information on the country, the year, the continent and the region.

# Order database
all_Merge <- all_Merge %>%
  select(code, year, country, continent, region, everything())

write.csv(all_Merge, file = here("scripts","data","all_Merge.csv"))

Here are the first few lines of the final dataset:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

Final structure of our merged database: each country of the 166 countries from D1_1_SDG are observed each year from 2000 to 2022, thus each row has a key composed of (code, year) that uniquely identifies an observation. The other columns are the variables listed above. Due to some countries having a lot of missing information we will have to eliminate some of them, but we will still have more than 2000 rows in our database.

Treatment of missing values

We load our final database and subset it according to the data that we will need in order to answer the different questions. This will help us dealing with the missing values.

# Import merged database
all_Merge <- read.csv(here("scripts","data","all_Merge.csv"))

For question 1, we only keep the years until 2020, because most of the explanatory variables that we want to use (those coming from the human freedom index) only have values until 2020.

# subset of data
# for question 1: factors (only until 2020 because no information for freedom index after)
data_question1 <- all_Merge %>% filter(year<=2020) %>% select(-c(total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages, cases_per_million, deaths_per_million, stringency, ongoing, sum_deaths, pop_affected, area_affected, maxintensity))

For question 2 and 4, we use the main data from the SDG database.

# for question 2 and 4: time and relationship between SDGs
data_question24 <- all_Merge %>% select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17))

For question 3, we create 3 distinct databases according to the different type of event that we wwill analyse: disasters, COVID19 and conflicts. For the disasters, we only keep the years until 2021, because after this date, we don’t have data. For the conflicts, we only keep the years until 2016, because after this date, we don’t have data.

# for question 3: events
# Disasters (only until 2021 because no information for disasters after)
data_question3_1 <- all_Merge %>% filter(year<=2021) %>% select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages))
# COVID
data_question3_2 <- all_Merge %>% select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, cases_per_million, deaths_per_million, stringency))
# Conflicts (only until 2016 because no information for conflicts after)
data_question3_3 <- all_Merge %>% filter(year<=2016) %>% select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, ongoing, sum_deaths, pop_affected, area_affected, maxintensity))

##### Which countries have many missing observations over the different variables of the different subsets?

Data for question 1

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. We decide to remove the countries that have more than 50 missing values.

#### Question1 
see_missing1_1 <- data_question1 %>%
  group_by(code) %>%
  summarise(across(-c(X, year, country, continent, region, population, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17), 
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 50))
# Remove countries where num_missing >= 50 ??
data_question1 <- data_question1 %>% filter(!code %in% see_missing1_1$code)

# List of countries deleted
list_country_deleted <- c(unique(see_missing1_1$code))

Here is the dataframe that allows us to see the countries that have missing values, how many and for which variables, when there are more than 50 in total.

code	unemployment.rate	GDPpercapita	MilitaryExpenditurePercentGDP	MiliratyExpenditurePercentGovExp	internet_usage	hf_score	pf_law	pf_security	pf_movement	pf_religion	pf_assembly	pf_expression	pf_identity	pf_score	ef_government	ef_legal	ef_money	ef_trade	ef_regulation	ef_score	num_missing
BHS	0	0	21	21	0	0	14	0	0	0	0	0	0	0	0	0	0	0	0	0	56
BTN	0	0	21	21	0	13	0	0	0	0	0	0	0	13	0	0	13	13	10	13	117
COM	0	0	21	21	3	19	0	0	0	0	0	0	0	19	19	19	19	19	19	19	197
CPV	0	0	0	0	0	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	60
DJI	0	0	15	12	0	19	0	0	0	0	0	0	0	19	19	19	19	19	19	19	179
GIN	0	0	7	7	0	13	0	0	0	0	0	0	0	13	0	0	13	13	11	13	90
GMB	0	0	0	0	0	10	0	0	0	0	0	0	0	10	2	0	10	10	10	10	62
IRQ	0	0	4	4	2	16	0	0	0	0	0	0	0	16	3	0	16	16	16	16	109
KHM	0	0	0	0	3	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	63
LAO	0	0	7	7	0	14	0	0	0	0	0	0	0	14	0	0	14	14	13	14	97
LBN	0	0	0	0	0	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	60
LBR	0	0	0	0	2	14	0	0	0	0	0	0	0	14	0	0	14	14	10	14	82
QAT	0	0	12	12	0	10	0	0	0	0	0	0	0	10	7	0	10	10	10	10	91
SAU	0	0	0	0	0	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	60
SDN	0	0	5	5	9	16	0	0	0	0	0	0	0	16	0	0	16	16	16	16	115
SOM	0	13	21	13	4	19	0	0	0	0	0	0	0	19	19	19	19	19	19	19	203
SUR	0	0	21	21	0	10	0	0	0	0	0	0	10	10	5	0	10	10	10	10	117
SWZ	0	0	0	21	3	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	84
TJK	0	0	4	5	3	10	0	0	0	0	0	0	0	10	0	0	10	10	10	10	72
YEM	0	1	5	5	3	10	0	0	0	0	0	0	0	10	10	0	10	10	10	10	84

Now, looking at the remaining countries that have missing values and there number accross all variables, we decide to remove MilitaryExpenditurePercentGovExp, because it has too many missing values and it contains similar information to MilitaryExpenditurePercentGDP.

see_missing1_2 <- data_question1 %>%
  group_by(code) %>%
  summarise(across(-c(X, year, country, continent, region, population, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17),
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))

# Delete MilitaryExpenditurePercentGovExp because it has too many missing values and contains similar information to MilitaryExpenditurePercentGDP

data_question1 <- data_question1 %>% select(-MiliratyExpenditurePercentGovExp)

Here is the dataframe that allows us to see the countries that have missing values, how many and for which variables, after remoying the countries with more than 50.

code	unemployment.rate	GDPpercapita	MilitaryExpenditurePercentGDP	MiliratyExpenditurePercentGovExp	internet_usage	hf_score	pf_law	pf_security	pf_movement	pf_religion	pf_assembly	pf_expression	pf_identity	pf_score	ef_government	ef_legal	ef_money	ef_trade	ef_regulation	ef_score	num_missing
AGO	0	0	0	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	30
ARE	0	0	6	6	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	12
ARM	0	0	0	0	0	4	0	0	0	0	0	0	0	4	0	0	4	4	3	4	23
AUS	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
AZE	0	0	0	0	2	4	0	0	0	0	0	0	0	4	0	0	4	4	0	4	22
BDI	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
BFA	0	0	0	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	30
BIH	0	0	2	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	32
BLZ	0	0	0	0	3	0	13	0	0	0	0	0	0	0	0	0	0	0	0	0	16
BRB	0	0	21	21	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	45
CAF	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
CIV	0	0	0	21	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	22
COD	0	0	0	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	21
COG	0	0	6	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	9
CRI	0	0	21	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	42
ETH	0	0	0	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	30
FJI	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2
GEO	0	0	0	0	0	2	0	0	0	0	0	0	0	2	0	0	3	2	0	2	11
GUY	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
HTI	0	0	13	13	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	26
ISL	0	0	21	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	42
JAM	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2
JOR	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2
KAZ	0	0	0	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	30
KGZ	0	0	0	0	0	5	0	0	0	0	0	0	0	5	1	0	5	5	5	5	31
LKA	0	0	0	0	6	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	6
LSO	0	0	0	0	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	30
MDA	0	0	0	0	3	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	33
MDG	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2
MKD	0	0	0	0	0	3	0	0	0	0	0	0	0	3	0	0	3	3	0	3	15
MMR	0	0	6	6	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	13
MNE	0	0	0	0	4	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	34
MNG	0	0	0	0	4	4	0	0	0	0	0	0	0	4	0	0	4	4	0	4	24
MOZ	0	0	0	0	0	3	0	0	0	0	0	0	0	3	0	0	3	3	0	3	15
MRT	0	0	0	7	0	5	0	0	0	0	0	0	0	5	0	0	5	5	5	5	37
MWI	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
NER	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
PAK	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
PAN	0	0	21	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	42
PNG	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
RWA	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	2
SRB	0	0	0	0	4	5	0	0	0	0	0	0	0	5	2	0	5	5	5	5	36
SYR	0	0	10	10	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	20
TCD	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2
TGO	0	0	5	5	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	10
TTO	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3
USA	0	0	0	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	21
VEN	0	5	0	21	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	29
VNM	0	0	5	5	0	2	0	0	0	0	0	0	0	2	0	0	3	2	0	2	21
ZWE	0	0	3	8	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	11

GDP per capita

Only Venezuela has missing values that we can not fill, so we delete the country.

# GDPpercapita
question1_missing_GDP <- data_question1 %>%
  group_by(code) %>%
  summarize(NaGDPpercapita = mean(is.na(GDPpercapita)))%>%
  filter(NaGDPpercapita != 0)
# Only VEN, we can't fill the missing, we delete VEN
data_question1 <- data_question1 %>% filter(code!="VEN")

# Update list countries deleted
list_country_deleted <- c(list_country_deleted, "VEN")

Military expenditure in % of GDP

To begin with, we delete the countries with more than 30% missing values.

# Military expenditure in % of GDP
question1_missing_Military <- data_question1 %>%
  group_by(code) %>%
  summarize(NaMilitary = mean(is.na(MilitaryExpenditurePercentGDP)))%>%
  filter(NaMilitary != 0)
# Remove the countries with more than 30% missing
data_question1 <- data_question1 %>% filter(code!="BRB" & code!="CRI" & code!="HTI" & code!="ISL" & code!="PAN" & code!="SYR") 

# Update list countries deleted
list_country_deleted <- c(list_country_deleted, "BRB", "CRI", "HTI", "ISL", "PAN", "SYR") 

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values, where there are less than 30% missing using the median by region.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
question1_missing_Military <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>%
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

# See the distribution of the missings per region
Freq_Missing_Military <- ggplot(data = question1_missing_Military) +
  geom_histogram(aes(x = MilitaryExpenditurePercentGDP, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of Military expenditures in % of GDP", x = "Military expenditures in % of GDP", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 3)

print(Freq_Missing_Military)

# All are skewed distributions, we decide to replace the missing values where there are less than 30% missing by the median by region

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(MilitaryExpenditurePercentGDP))
  ) %>%
  ungroup() %>%  # Remove grouping temporarily
  group_by(region) %>%
  mutate(
    MedianByRegion = median(MilitaryExpenditurePercentGDP, na.rm = TRUE),
    MilitaryExpenditurePercentGDP = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(MilitaryExpenditurePercentGDP),
      MilitaryExpenditurePercentGDP,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, MilitaryExpenditurePercentGDP)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

Internet usage

There are only low percentage of missing values.

# Internet usage
question1_missing_Internet <- data_question1 %>%
  group_by(code) %>%
  summarize(NaInternet = mean(is.na(internet_usage)))%>%
  filter(NaInternet != 0)

# Only low % of missing

We look at the evolution of the variable over time. We fill the missing values with linear interpolation, because all evolutions are in an increasing way and are almost straight lines, except for CIV that we delete.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
question1_missing_Internet <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(internet_usage))) %>%
  filter(code %in% question1_missing_Internet$code)

# Look at the evolution over the years for the countries that have missing values
Evol_Missing_Internet <- ggplot(data = question1_missing_Internet) +
  geom_point(aes(x = year, y = internet_usage, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of internet usage over time", x = "Years from 2000 to 2022", y = "Internet usage") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  scale_x_continuous(breaks=NULL)+
  facet_wrap(~ code, nrow = 4)

print(Evol_Missing_Internet)

# Fill with linear interpolation, because all evolution are in an increasing way and are almost straight lines, except for CIV
list_code <- setdiff(unique(question1_missing_Internet$code), "CIV")
for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- data_question1 %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$internet_usage)
  
  # Update the original dataset with the interpolated values
  data_question1[data_question1$code == i, "internet_usage"] <- interpolated_data
}

# Delete country CIV
data_question1 <- data_question1 %>% filter(code!="CIV")

# Update list countries deleted
list_country_deleted <- c(list_country_deleted, "CIV") 

Human freedom index

First, we remove hf_score, pf_score and ef_score, because there are many missing values and since these variables summarize the other ones, deleting the will not make us loose information.

# Human Freedom Index
# Remove hf_score, pf_score and ef_score because many missing and since these variables are summary of other ones, deleting the mwon't make us loose information
data_question1 <- data_question1 %>% select(-c(hf_score, pf_score, ef_score))

# pf_law has (many) missing only for one country:BLZ, we remove it 
data_question1 <- data_question1 %>% filter(code!="BLZ")

# Update list countries deleted
list_country_deleted <- c(list_country_deleted, "BLZ") 

Economic freedom: government

Only KGZ and SRB have missing values, we plot the values over time and fill in the missing values by the year before, since there are only one and two missing values.

# ef_government: KGZ and SRB have missing values -> plot
# KGZ
Evol_Missing_ef_gov <- data_question1 %>% group_by(code) %>% filter(code=="KGZ")
ggplot(Evol_Missing_ef_gov, aes(x = year, y = ef_government)) +
  geom_point() +
  labs(title = "Evolution of economic freedom: government over time in KGZ", x = "Years", y = "ef_gov")
# Only one missing, in 2000, replace by the value of 2001
# SRB
Evol_Missing_ef_gov <- data_question1 %>% group_by(code) %>% filter(code=="SRB")
ggplot(Evol_Missing_ef_gov, aes(x = year, y = ef_government)) +
  geom_point() +
  labs(title = "Evolution of economic freedom: government over time in SRB", x = "Years", y = "ef_gov")
# Only 2 missing, replace by next value
data_question1 <- data_question1 %>%
  mutate(ef_government = ifelse(code == "KGZ" & year == 2000 & is.na(ef_government), ef_government[which(code == "KGZ" & year == 2001)], ef_government))
data_question1 <- data_question1 %>%
  mutate(ef_government = ifelse(code == "SRB" & year == 2000 & is.na(ef_government), ef_government[which(code == "SRB" & year == 2002)], ef_government))
data_question1 <- data_question1 %>%
  mutate(ef_government = ifelse(code == "SRB" & year == 2001 & is.na(ef_government), ef_government[which(code == "SRB" & year == 2002)], ef_government))

Economic freedom: money

18 countries have missing values, but the percentage of missing values is always below 25%.

# ef_money
question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_ef_money = mean(is.na(ef_money)))%>%
  filter(Na_ef_money != 0)
# All below 25%

We look at the evolution of the variable over time. For the countries where this evolution is linear, we fill in the missing values using linear interpolation.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_money))) %>%
  filter(code %in% question1_missing_ef_money$code)

# Look at the evolution over the years for the countries that have missing values
Evol_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
  geom_point(aes(x = year, y = ef_money, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of economiv freedom: money over time", x = "Years from 2000 to 2022", y = "ef_money") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 4) +
  scale_x_continuous(breaks = NULL)

print(Evol_Missing_ef_money)

# Linear interpolation for "ARM", "BFA", "BIH", "GEO", "KAZ", "LSO", "MDA", "MKD"
list_code <- c("ARM", "BFA", "BIH", "GEO", "KAZ", "LSO", "MDA", "MKD")
for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- data_question1 %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$ef_money)
  
  # Update the original dataset with the interpolated values
  data_question1[data_question1$code == i, "ef_money"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values using the median by region.

# Create a dataframe that only have the countries with missing values and add a column which contains the % of missings for each country
question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_money))) %>%
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

# See the distribution of the missings per region
Freq_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
  geom_histogram(aes(x = ef_money, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of economic freedom: money", x = "ef_money", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 3)

print(Freq_Missing_ef_money)

# All are skewed distributions, we decide to replace the missing values where there are less than 30% missing by the median by region

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(ef_money))
  ) %>%
  ungroup() %>%  # Remove grouping temporarily
  group_by(region) %>%
  mutate(
    MedianByRegion = median(ef_money, na.rm = TRUE),
    ef_money = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(ef_money),
      ef_money,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, ef_money)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

Economic freedom: trade

19 countries have missing values, but the percentage of missing values is always below 25%.

# ef_trade
question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_ef_trade = mean(is.na(ef_trade)))%>%
  filter(Na_ef_trade != 0)
# All below 25%

question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_trade))) %>%
  filter(code %in% question1_missing_ef_trade$code)

We look at the evolution of the variable over time. For the countries where this evolution is linear, we fill in the missing values using linear interpolation.

# Look at the evolution over the years for the countries that have missing values
Evol_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
  geom_point(aes(x = year, y = ef_trade, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of economic freedom: trade over time", x = "Years from 2000 to 2022", y = "ef_trade") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 4) +
  scale_x_continuous(breaks = NULL)

print(Evol_Missing_ef_trade)

# Linear interpolation for "AZE", "BFA", "ETH", "GEO", "VNH"
list_code <- c("AZE", "BFA", "ETH", "GEO", "VNH")
for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- data_question1 %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$ef_trade)
  
  # Update the original dataset with the interpolated values
  data_question1[data_question1$code == i, "ef_trade"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values using the median by region.

# Create a dataframe that only have the countries with missing values and add a column which contains the % of missings for each country
question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_trade))) %>%
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

# See the distribution of the missings per region
Freq_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
  geom_histogram(aes(x = ef_trade, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of economic freedom: trade", x = "ef_trade", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 3)

print(Freq_Missing_ef_trade)

# All are skewed distributions, we decide to replace the missing values where there are less than 30% missing by the median by region

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(ef_trade))
  ) %>%
  ungroup() %>%  # Remove grouping temporarily
  group_by(region) %>%
  mutate(
    MedianByRegion = median(ef_trade, na.rm = TRUE),
    ef_trade = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(ef_trade),
      ef_trade,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, ef_trade)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

Economic freedom: regulation

12 countries have missing values, but the percentage of missing values is always below 25%.

# ef_regulation
question1_missing_ef_regulation <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_ef_regulation = mean(is.na(ef_regulation)))%>%
  filter(Na_ef_regulation != 0)
# All below 25%

We look at the evolution of the variable over time. For the countries where this evolution is linear, we fill in the missing values using linear interpolation.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
question1_missing_ef_regulation <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_regulation))) %>%
  filter(code %in% question1_missing_ef_regulation$code)

# Look at the evolution over the years for the countries that have missing values
Evol_Missing_ef_regulation <- ggplot(data = question1_missing_ef_regulation) +
  geom_point(aes(x = year, y = ef_regulation, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of economic freedom: regulation over time", x = "Years from 2000 to 2022", y = "ef_regulation") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 4)

print(Evol_Missing_ef_regulation)

# Linear interpolation for "ETH", "KAZ", "MDA", "SRB"
list_code <- c("ETH", "KAZ", "MDA", "SRB")
for (i in list_code) {
  # Filter the dataset for the current country
  country_data <- data_question1 %>% filter(code == i)
  
  # Perform linear interpolation for the current country's data
  interpolated_data <- na.interp(country_data$ef_regulation)
  
  # Update the original dataset with the interpolated values
  data_question1[data_question1$code == i, "ef_regulation"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values using the median by region.

# Create a dataframe that only have the countries with missing values and 
# add a column which contains the % of missings for each country
question1_missing_ef_regulation <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_regulation))) %>%
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

# See the distribution of the missings per region
Freq_Missing_ef_regulation <- ggplot(data = question1_missing_ef_regulation) +
  geom_histogram(aes(x = ef_regulation, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 100) +
  labs(title = "Distribution of economic freedom: regulation", x = "ef_regulation", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 3)

print(Freq_Missing_ef_regulation)

# All are skewed distributions, we decide to replace the missing values where there are less than 30% missing by the median by region

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(ef_regulation))
  ) %>%
  ungroup() %>%  # Remove grouping temporarily
  group_by(region) %>%
  mutate(
    MedianByRegion = median(ef_regulation, na.rm = TRUE),
    ef_regulation = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(ef_regulation),
      ef_regulation,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, ef_regulation)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

We visualize the NAs for question 1 to make sure we did not miss any and we see that except for goals 1 and 10 that were already discussed, and we are surprised to see that even these variables don’t have missing values anymore. Indeed, by removing some countries according to the number of missing values or difficulty to fill them, we removed the countries that were problematic for goals 1 et 10.

#### Visualization NA Q1 ####
na_counts <- data_question1 %>%
  group_by(code) %>%
  summarise(across(-c(X), ~ sum(is.na(.)))) %>%
  mutate(num_missing = rowSums(select(., where(is.numeric)))) %>%
  filter(num_missing == 1)

print(na_counts)
#> # A tibble: 0 x 40
#> # i 40 variables: code <chr>, year <int>, country <int>,
#> #   continent <int>, region <int>, population <int>,
#> #   overallscore <int>, goal1 <int>, goal2 <int>, goal3 <int>,
#> #   goal4 <int>, goal5 <int>, goal6 <int>, goal7 <int>, goal8 <int>,
#> #   goal9 <int>, goal10 <int>, goal11 <int>, goal12 <int>,
#> #   goal13 <int>, goal15 <int>, goal16 <int>, goal17 <int>,
#> #   unemployment.rate <int>, GDPpercapita <int>, ...
######## NO MORE MISSINGS FOR QUESTION 1

Data for question 2 and 4

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.

#### Questions 2 and 4
see_missing24 <- data_question24 %>%
  group_by(code) %>%
  summarise(across(everything(), ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))
# Nothing to remove, only goals 1 and 10 have missing (already discussed before)

Data for question 3

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.

Disasters

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. We find out that there are many missing values… Sofia ? :)

#### Question 3

# Disasters
see_missing3_1 <- data_question3_1 %>%
  group_by(code) %>%
  summarise(across(-c(goal1, goal10),  # Exclude columns "goal1" and "goal10"
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))
# Many missing, what do we do?

COVID19

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.

# COVID
see_missing3_2 <- data_question3_2 %>%
  group_by(code) %>%
  summarise(across(-c(goal1, goal10),  # Exclude columns "goal1" and "goal10"
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))
# No missing

Conflicts

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed.Two countries have missing values, we remove them (MNE and SRB).

# Conflicts
see_missing3_3 <- data_question3_3 %>%
  group_by(code) %>%
  summarise(across(-c(goal1, goal10),  # Exclude columns "goal1" and "goal10"
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))
# 2 countries have missings, we remove them: MNE and SRB
data_question3_3 <- data_question3_3 %>% filter(!code %in% c("MNE","SRB"))

##### EXPORT as CSV #####
write.csv(data_question1, file = here("scripts","data","data_question1.csv"))
write.csv(data_question24, file = here("scripts","data","data_question24.csv"))
write.csv(data_question3_1, file = here("scripts","data","data_question3_1.csv"))
write.csv(data_question3_2, file = here("scripts","data","data_question3_2.csv"))
write.csv(data_question3_3, file = here("scripts","data","data_question3_3.csv"))